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of items from the total set of test Items, and use item response theory^ 
with each subset. While factor analysis is the most commonly proposed pro- 
cedure for dalerinining dimensionality, a recently developed procefiure called 
order analysis iu^y also fprove to be useful for isolating unidimens jjonal 
iteto^sets-. ' * 

The ftrs-t study in this report dealt with' a comparison of three ordet , 
analysis procedures:. Kxfus,& Bart'^ (1974) procedure and Reynolds' (1976) 

and^ c^ 



t3' 

tasets with 



procedures using twq of Cliff's- (1977)' consistency indices, c 
respectively, -The comparisons were based on 'seven simulated da 
known factorial dimensionality, and two multidimensional setsT of mathematics 
data^ The ,c^ prcfcedure regr9duced th^- factor structure for all of the 
, Simula t-ed datasets, while the other twi procedures performed yery poorly. 
'However, for the mathematics cfata, all, three procedures failed to repro-^ ' 



duce the factors. 



T^le seconcT^tudy in this report pi:esents preliMtlary results us^ng a new 
order-analysis procedure which solves some of the dif f icu}.ties with 'the 
othep procedures in reproducing factorial dimensionality, iThis new proce- 
dure '(dubbed ORDO) Reproduced the factors for the mathematics data as well 
as for tt|e simulated data. It is hoped «that ORDO will represent a useful 
alternative to factor analysis for determining uni4iniensional ^tem sets 
appropriate for J.atent-tralt methods. 
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* Steven* L. vJi 



se 



* ♦ * ABSTRACT . ' 

Cu^Tent latent-trait methods require that the latent space" 
underlying a group test .performi'nce be unidi'mensional. Ho\>ever, 
many tests- yield multidimensloifal dkta, implying that more than one 
latent trait wourd be necessary to account* for -'test performance. A 
ppssible solution to this problem of ,multJ.dimens'ionality would be to 

> . ; - ' \ ' 

isolate unidimensional subsets of items from the, total set of test 
items and. use item re^onse theory witsfe each subipet;. While factor 
analysis is the most commonly prgpdsed piocedure/for determining . 

dimensionaltiy, a recently developed procedtire/c^lled order Analysis 

\ J / 

may also prove to be useful for isolating ahidii^iensional item sets. 

' ' . V 1, \ • 

The first study in this report dealt with a comparison of three 
order analysis procedures: Krus & Bart's (1^/4) procedure and ■ 
Reynolds' (1976) procedures using two of Cli^,^'^ (1977) consistency 



indices, c . and c , resbecfively . . The comparisons were based on 
sefVen simulated data^ets With known factorial dimensionalit;y, and 
two multldimenaioi il set^ of mathematics dafa. Th^^^^ procedure 
reproduced the factor structure for* all of che simulated datasets. 



while the other two procedures performed very poorly. However, for the 
mathematics data, all three procedures failed to reproduce the factors. 

Tl^ second study ia this repoi't presents preliminary results 
using a new order-analysis procedure which solves some of the diffi- 
culties with the other procedures in repro(fucing factorial dimension- 
ality. This new procedure (dubbed (JrdO) reproduced^ the factors for 
the mathematics data as well as for the simulated data. It is hoped 
that ORDO will represent a useful alternative to factor analysis for 
dettonining unidimensional itei^ets appropriate for latent-trait 
methods, ^ . . . 
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^ • Introduction 

A major Issue in item response theory concerns determining -tjke 
number of latent dimensions (traits) needed to adequately account fo'r • 
the test performance of a group of individuals. If. all of the relevant 
dimensions are not accounted for, thenjfche requirement of local 
independence of items will not hol5 and the it^e^i response model' will 
be intractable. This problem is compounded by current practical 
limitations of item 'response theory. While there have been multidimen-^ 
stonal latent trait models proposed, ' estimation problems arising 
from these models h^ve rendered them all but useless in the field. 
Hence, the current ^tate of affairs regarding item respoi^e theory ' 
prevents one from considering more than one latent tra^t at a time. 
This means th<^t the latent space, under consideration has to be uni- 
diipensional in order to be practicable. However, many tests yield 
multidimensi^al ddtd, implying that more than one latent trait would 
be necessary to account for test performance. 

One possible solution to this probTem of multidimensionality 
would be to extract unidimensional subsets of items from the larger, 
multidimensional^^^f of items, and useyi^tei? response theory to generate 
^ ^ separate ability estimates from each subset. The most commonly pre-^ 
scribed method of determining the dimensionality of a set of- items is 
factor Analysis. However, Krus (1975) points out that factor analysis 
methods contain a considerable amoi^nt of indetermjnancy due to a 
relative lack of consensus ^regarding such issues as (I)' appropriate 
factor extraction method, (2) the problem of communality estimation, 
and (3) the number />f factors to extract. Krus .has Sijggesteci use of 
order analysis as an alternative to factor analysis in determining 
the dimen8it>nality of a set of data. » ^ 

♦ Order analysis (Kru^s, Bart, & Airasian, 1975; Krus, 1975) was 

developed to investigate logical relations between the elements of 
a binary data matrix-. The method presumes that elements measuring ,* 
a single dimension show characteristics ,pf a strong simple order. 
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i.e., that the relations between the elements are transitive, asym- 
metric, and connected (see Coombs, Da^es, and Tversky, 1970). 

' The relation bf^ interest in order analysis is dominance . If* 
a .person » fails item i and passes item j, then item i is said to 
dominate item j for'that person. This follows from transitivity; 
since the persoh is dominated by /item i (fails item* i) and the person 
dominates item j (passes item j) then it is implied that item i 
dominates item j. .This will be callei an ij dominance. 

If there is a one-dimensional latent attribute underlying thfe 
behavior reflected by the data, then tH^ item relations will be 
consistent across persons (Coombs, et al., 1970). Her^ce, for any' 
items i and j, all persons should show either an ij dominance, or they 
should all show a ji dominance. Lack of consistency across persons 
is in violation of the order-analytic model. However, since there 
'ire usually errors ot measurement present in. the .data matrix, some • 
amount of inconsistency is tolerated. Krus et al., (1-975) proposed 
the^use of McNe^ar's (1947) £ statistic for correlated proportions ,to 
evaluate the preponderance of ij dominances over ji dominances. If 
^he value of is sufficiently large, then item i is concluded to ' 
dominate item' j for the entire group. ^ It is also assumed that the 
ji^dominances are due to error. In the case where there is a single 
order prfesent in the'set of items, e^ch item will dominate all items 
"below" it* in the order, and transitivity, asymmetry and connectedness 
will all be realizedX This set of items,'' also called a chain, will 
essentially form a Guttman scale. ^ 

There; are times, however, *when the value between two items i - 
and j does not indicate a clear ij dominance or ji dominance. This 
violates the connectedness pi|pperty that there mtiftt be a relation 
between each pair of items in the order. According |o Krus (1975), 
this indicates that items i and j are^^not members of the .same order, 
'and that the data are multidimensio\ial. Based on this, a deterministic 
order-analytic model f »r determiiy.ng the dimensionality of an item 
set was developed (Krus & Bart^l^74), and^ later a probabilistic 
'model, (Krus, 1977). , * ' 



Cliff (1977) developed a r^umbe^ of indices to assess the consistency 

of simple orders. The first, c^>, reflects the proportion of the total 

number of t dominances in a d^taset which are consistent with a particular 

ordering. Another iioportant index, q^_, is similar to c , except that 

• , ^ tJr tl . 

it contains an adjustment for the number of dominances: expected 

♦ ^ 

chance fcr independent items". It i-s eopuiValent to Loevinger's (1947) 
index of homogeneity. 

Reynolds (1976) Rejected the approach of using Mckema'r'-s z 
test to evalu^t^ the relation between^ items and then using the relations 
to generate item chains. He pointed out- that Krus and Bart's (19^74) 
deterministic method does hot necessarily yield a^ique set of item 
-chains and that., other, more "opTiifaal'" chains may al^o be extracted. 
Reynolds also noted that the llrud and Bart procedure lacked any ^ 
goodness-^of-fit st-atistics to evafuate how well an ordering is consis- 
tent acrpss persons. Reynolds outlin^^^h alggrithm, usJLng one ot 
Oliff's (1977) -consistency indicea, to extract item chains. Each 
item In the set is used a's. a* starting podnt in a chain. The most ^ 

4 

consiste/it items. are then successively added , to the chain until the 
overall chain consistency index va!|^e drops below some minimally \ 
acceptable level. "Redundant chains arfe then deleted, and the remailiing 
chains are interpreted as representing the dimensions of the dataset. 

Earlier studies have" failed to show a consistent relationship" 
between the results of order anal/sis and factor tnalysis.' Krus and 
Weiss (1976) found congruence between thfe two methods for Thurs tone's 
1947, p. 140-143) "bQx datA". HcJ^ver, when they analyzed random 
data using ArmstrW and Soelberg's (1968) method, they found differing 
results using, order analysis and factor analysis. Bart (1978) rfeana- 
lyzed Che data reported in Bockv^d tiebeman (1970) and cohcluded * 
that- the factor structure o^ a set of data did not api)ear to relate in 
a clear way to the order structure. 
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Study I 

^ The purpose of the first, study was to compart dif f erent^rder- ' 
analysis procedures on a number of^ dar.asets with varying fatrtorlal 
dimensionality, Severf simulated dichotomous datasets were generated. 
These datasets differed both In terms of number of common factors arid ^ 
In terms of variance of th«<*item difficulty i4vels. Also, two datasets' 
con5)osed of signed-numbers mathematics i cems (described more, fully 
In Blrenbaum and Tatsuoka (]i980)) were used Ir^ comparing. the order- 
analysis procedures. xKese analyses could aid Iji the understanding 
of the differences amo,t\g the procedures, as well as providing insight 
regarding which procedure would be most useful In extracting sets of » 
items which satisfy the unldlmenslonality assumption of current latent- 
trait models (Lord & Novlck, 1968). ' * A 



Method ' ^ 

Simulated Datasets / ' • 

; . . 

Seven sfmulate^d dichotomous datasets were generated using the 
FORhWL^and TUCKLIB packag^ of -FORTRAN subroutines at Che University 
of Illinois. Each ^daraset, which consisted of 10 items and 500 
persons, was compulsed as follows. * A f^tor patCetn -matrix and a 
vector, of uniquenesses were specif ied^l^ the uset. ' From this infor- 
Mtlon a papulation variance-covari^nce ntatrix was g-enerated^ using 
a modified Tucker,* Koopman, and Linri (1969) procedure which 
simulated the effects of fandonuerror on the variance-covai:iance 
matrix by allowing I for the Influence of 'a number of minor* random 
factors.* This population variance-cfivariahce matrix was'^'then used in 
conjunction with a vector of user-specified, population item means to 
generate dichotomous. item scores *from a multivariate normal t)oPulatibn. 

The seven sitiuilated datasets are described In Table 1. It 
w48 decided that me distributions of item difficulty levels might have 



Table 1 



Descriptions of the^ Simulated Datasets 



Dataset Label* 



De§.cription (10 items, N=500) 



ni 


One 


ftactor with high spf&cing between the item Tneaps . 


Ml 


One 


factor with moderate s'JJacing between tj^ means. 


LI 


One 


factor. with low spacing between the itemr means. 


H2 


Two 


factors with high spacing between the item means. 


M2 


TW6 


* 

factors with moderate spacing between th& item meand. 


L2 


Two 


factors with low spacing between the item means. 


MIO 


Consisted of ten independent items with moderate spacing 
between the means (essentially a- lO^dimensional dataset)* 



Table 2 

Examples of jMe 16 Signed-Number 







llathematics Skills 




Item (Skill)' 




• • * 
A Example 


/ Operation 


1 




4 " C"1'0) = 11 


Subtractioji 


2 




•9 - (-7) - 16 


Subtraction 


3 




-7 - 9 = -16 


Subtraction 


4 


• 


-12 - 3 - -15 ' 


Subtraction 


5 




-3 - 12 - -15 


^Jl Subtraction 


• 6 




-6.- C-8) - 2 


Sub traction 


7 




-16 - (-7)* ^ -9 


Subtraction 


8 




8 - 6 « 2 


Subtraction 


9 




2 - 11 « -9 


* Subtraction 


* • 




6 + 4 « 10^ 


Addition 


ii 




-14 + (-5)*« -19 


' Addition 


12 




-5 + (^7) = -12 


Addition 


13 




-3 + 12 « 9 


Addition 


14 




-6 + 4 - -2 


' Addition 


15 




12 + (-3) = 9 


Addition 


16 




3'+ (-5) « -2 ^ 


Addition 



' 12 
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a differential effect on the order-analysis procedures. Hence, three 
types of item mean distributions wera used: Highly spaced means where 
each item difficulty level is very distinct from that of the other 
items, moderately spaced means where soire item diffjculty levels are 
sim-i.lar, and means which had the same population difficulty level 
but whose differences in sample difficulty levels were due only to 
random variation. ^\lso, for the two-tactor'^datasets (H2, M2, L2) 
items 1-4 always loaded on one factor, and items 5-10 loadsd on 
the other factor. 

Dataset MIO was unique in that it was generated so that there 
werfi no common factors among the items. It consisted of 10 unrelated 
items with moderately spaced means. This dataset was useful In 
comparing 'order-analysis "procedures in their abilities to indicate a, 
lack of order structure. 

Mathematics Data 

The mathematics dataset consisted of 16 dichotomous mastery 
score^>^rived from a 64-item signed-numbers test administered to 
125 eighth grade students during November, 1979. There were 16 
skills, each measured by f our^parallel^icems. Examples of these 
skills are 'thown in able 2. If a student^^get^^ajjeast three of the 
four items correct, he or she was deemed a master oE^^^felv^t skill and 
given a mastery score of one. Otherwise, a score of zero was'-glven 
(non-mastery) . 

Two forms of the mathematics dataset were analyzed. Birenbaum 
and Tatsuoka (1980) describe a procedure for detecting inapi opriate 
strategies used by students in solving signed-number problems. Often, 
students caii get ''correct" answers to some of these problems uding 
incorrect strategies. Once an incorre(:^ strategy was detected for 
a given student, it was possible to determine the items for which the 
stodent would have given the correct answer using -the inappropriatjjR 
sLtategy. An "adjusted** dataset was then construi^ted from the original 
64-item mathematics dataset such that items deemed to have been gotten 

c 

f 

a 



correct by an inappropriate strategy were rescored as incorrect. 
Dichotomous mastery scores were then recomputed for the adjusted 
dataset/ Order analyses were subsequently performed on both the 
unadjusted (UMATH) and adjusted (AMATH) 16-item mastery datasets. 
« 

Order-Analysis Procedures 

Three order-analysis procedures were used: the deterministic 
order-analysis method of Krus and Bart (1974), Reynolds' (1976) 
algorithm using c^^ as an extraction index, and Reynolds' procedure 
using c^^. To determine the presence of a relation in Krus and 
Bart's procedure, a criterion McNemar's z value of 1.64 was used. 
Krus' (1977) probabilistic order-analysis procedure was not used for 
'two reasons. First, it was decided that the results obtained from 
the deterministic and probabilistic models would be similar enough 
that both procedures' would not be necessary in this study. Second, 
since Reynolds' (1976) method is deterministic, the deterministic 
order-analysis method was chosen in ordfer to permit the most straight- 
forward comparisons" among the results of the different methods. 



Results 



Simulated Data 

In ord^ o verify the factor structures of the simulated 
datasets, common factor analyses of the matrices of phi 

coefficients were performed. For datasets where more than one common 
factor was extracted, factors were rotated using the Varimax criterion 
The results of these factor ai\alyses, along with the item means and 
standard deviations, are shown in Appendices 1 through 7. All seven 
datasets Uhowed clear factorial dimensionality in agreement with the 
factor pa:ttern matrices , from which the datasets were generated; For 
dataset mIo, a scree test of the eigenvalues led to the conclusion ^ 
that no common factors were present. 



Table 3 



Item Chain Extraction for Datasets HI, Ml, and LI 

i . , 

1^ 

Item Chains Extracted / 



Krus & Bart ^tl ♦ * ^t3 

Dataset Procedure Procedure ^ Procedure 



HI (1-2-3-4-5-6-7-8-9-10) (1-2-3-4-5-6-7-8-9-10) (1-2-3-4-5-6-7-8-9-10) 

Ml (2-3-4-5-6-7) (1-2-3-4-5-6-7-8-9-10) (1-2-3-4-5-6-7-8-9-10) 

(1) 



(8) 
(9) 
(10) 



(1) (1) (1-2-3-4-5-6-7-8-9-10) 

(2) C2) 
O) (3) 

(4) (4) 

(5) C5) 

(6) C6) 
■ (7) (7) 

(Sy (8) 

(9) (9) 

(10) CIO) 



Note: ^utoff values o£ and c^^ "sed were .90 and. .70, respectively, 
I - All of the Items loaded on a single factor. 



1 



The order-analysis results for datasets HI, Ml, and LI are 
shown in Table 3. For Hl^ ail three procedures correctly extracted 
a single chain (dimension) of itejns. For Ml, the three procedures were 
not in agreement. While the sii7gle chain was correctly extracted 
using c^^, use of the other two' procedures yielded multiple chains. 
However, if the minimum consistency level of c is lowered from ,90 
to .86, then che correct single chain would, hav^ been extracted for 
the^c^^ procedure. For dataset LI, composed of items which were 
highly similar in terms of difficulty level, the c^^ procedure was the 
only procedure which extracted the .single dimension. The other. two 
procedures failed to determine any item chains. Note that the overall 
value of c^^ was near zero, while for c^^j it was fairly high.^ 

Table 4 shows, the order-analysis results for datasets H2, M2, 
and L2. For K2 Krus & Bart's (197^ procedure coUld not accurately 
extract the two- factors. Items 6, 7, and 9 were incorrectly ^ombined 
in a chain with items 1, 2, 3, and 4. Reynolds' procedure extracted 
the correct chains whei;i eit;her c^^ or c^^ was used. For M2, however, 
only the c^^ procedure extracted the two dimensions. The c^^ 
procedure extracted one of the dimensions, 'but could not extract the 
other. The results for Krus and Bart's procedure were chaotic in 
terms of the factor structure of this dataset.' For dataset L2, 
as for LI, only the c^^ procedure^ .correctly extracted the two 4imensioPS 

I The other two proctedures failed to combine any items Into chains* 
r The chain extractio* results* for MIO, shown in Table 5, 
pilustrated other differences among tfie three procedures.. In this 
dataset, there were no real common factors present among the items. 
The c^2 procedure extracted no chains at all. Krus and Bart's 
procedure, however, yielded a large (8-item> chain, and the 
c^j procedure yielded a number of small chains. 



Table 4 ' . 
Item Chain Extraction for Datasets'fK, M2, and L2 



Dataset 



Kriis & Bart 
Procedure 



H2 (1-6-2-7-3-9-4) 
. <'5-8-10) 



Item Chains Exrracted 



1 



^tl 
Procedure 



(1-2-3-4) 
(5-6-7-8-9-10) 



"t3 
Procedure 



(1-2-3-4) 
(5-6-7-8-9- rO) 



Overall Consistency 
Statistics , 



c . c^, KR20 



.655 .393 .749 



M2 



LI 



C6- 1-7-4-9) 
(5-8-10) . 
C2) 
C3) 

r 

CD 
C2) . 
C3) 
(4) 

(5) . • 

C6)- . 

C7) 

(8) 

(9) 

(10) 



a- 3-4-9) ' 
(1-2-4-9) 
(5-6-7-8-9- 10 V 



CD 

(2) 

(3) 

04) 

(5)' 

(6) 

C7) 

C8) 

C9) 

(10) 



(1*2-3-4) 
(5-6^7-8-9-10) 



0-2-3-4) ■ 
(5-6-7-8-9-10) 



< .755 .374 .703 



.036 .329 .826 



« Note: Cutoff values of c^.^ and c^.^ used were ,90 and .70, respectively. 

- 1 - Items 1 - 4 loaded- highly on om factor, and Items 5 - 10 loaded highly on the other factor. 
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, Item Chain Extraction for. Dataset MIO 



Dataset 



Kr-us & Bart 
Procedure 



MIO (l_2-3-6-7-8-9-Jl0), 
(5) 



Item Chains Extracted^- 



"tl - 
Procedure 



t3 

Procedure 



(1-2E-10) 
(1-3) ^ 

(1-8J 
(1-9) 
14) 
(5) . 
(6) 



(1) 

(2) 

(3) • 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

(10) 



Note: Cutoff values of c^^ and c^^ used were .90 and .70, respectively. 

* ' r 

1 - This dataset contained no coiumon factors. - 



Overall' Consistency 
Statisxics 



\\ c^3 .KR20 



.606 -.011 -.057 
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Mathematics Data . . ^ 

Factor analyses of the matrices of phi coefficients for the two . 
mathematics datasets afre shown In Tables 6 and 7, For the UMATH 
^ dataset, two-factor solution is presented, although a scree test of 
the eigenvalues did not clearly suggest the number of factors to 
extr-act. Factor solutions were obtained for two through five factors, 
and the two-factor solution best approximated simple structure. *^The - 
subtraction items (1 - 9) comprised one factor, while four of the 
addition items (13 - 16) comprised the second factor. The four 
second-factor items were all skills dealing with the "addition of 
two numbers that wer^ opposite in sign. 

However, when the data were adjusted for presumably errone^sly 
correct responses (AMATH) , two clear factors of subtraction and 
addition emerged. Only two eigenvalues were greater thSp one, and 
a very c'fear simple structure was present. The correlation between • 
the two factors was ,46. . ' • t 

Order analyses of the matheipat;lcs data, shown in TaBle A, gave 
very different results from those of the factor analyses. For both* 
datasecs, neither the Krus & Bart procedure nor the c^^ procedure 
yielded chains that showed any resemblance to the "factors. The 
procedure also failed to reproduce the factor structure for 
either dataset. For AMATH In particular, the c ^ procedure found 
one chain with fairly high overall consistertcy (c^^ " -764). 



Discussion 



It quickly became clear from the results^ of tKe simulated data 
that Krus & Bart's (1974) procedure did not perform very well, in 
reproducing the factor structures of the datasets. The c^^ procedure 
did not fare much better; it reproduced the factox structures only for 
datasets HI and H2. Basically there ^re two reasons for the poor 
results from these tWo procedure;^. First,* when a factor contains two 
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Table 6 ,.- 

Simple* Common Factor Analysis of Phi ■ t 
Coefficients for the Unadjusted Mathematics (UMATH) Dataset 



Item 



1 
2 
3 
4 
5 
6 
7 
8 
.9 
10 
11 
12 
13 
14 
15 
16 



Mean 



S.D. 



Factor I 
loadings 



.648 

.6 80' 

.584 

.576 

,720 

.744 

.824 

.856 

.704 

.992 

.912 

."920 
.944 
.920 
.^20 



.480 
.468 
.495 
.496 
.451 
.438 
^382 
.352 
.458 
.089 
■ .284 
.246 
.272 
.231 
.272 
.272 



.817 
.847 
.696 
.698 
.891 
.713" 
.617 
.485 
.633 
.119 
.368 
.352 
.038 
-.011 
-.035 
.051 



Fc^ftor II 
loadings 



-.119 
-.120 
-.078 
-.162 
-.096 
.082 
.068 
-.025 
.118 
.037 
.159 
.060 
.765 
.591 
.509 
.684 



Eigenvalues 



5.449 
2.245 
1.665 
1.223 
1.028 
.757 
.712 
.545 
; .456 
.407 
,.371 
.338 
.252 
.238 
.165 
.150 



Note:, Factors were rotated using the Oblimin method. 
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Table 7 



Simple Common Factor Analysis > of » Phi 
Caef ficilnts^or the Adjusted Mathematics ^(AMATH) 



Dataset 



ERIC 



Item 


Mean 


S.D. 


Factor I 
loadings 


Factor ^I 
loadings 


Eigenvalues ' 


— r 


.600 










1 ■ 


.492. 


.834 


.015 


8.216" 




.624 


.486 


.865 


.007 


2.881 


^3 


.536 


.501 


.771 


-.022 0 


.^36 


A 


...536 


.501 


.770 


-.024 


.682 


5 


.688. 


.465 


.937 


.030 


.608 


6"* 


' .664 


.474 


.895 


.041 


.501 


' 7 


.696 


.462 


- .920 


.050 


.396 


8 


.792 


.408 


.679 


-.068 


.370 


9 


.648 


.480 


.749 


.068 


, ' .347 


10 - 


.960 


.197 


.144 


.353 


.285 


11 


.888 - 


.317 


. 108 


.745 


.251 - 


12 4 


.904 


.296 


.069 


.786 


.225 


13^ 


.888 


•^'] 


-.086 


.858' . 


.172 


14 •• 


.896 




-.008 


.850 


.099 


15 


.872- 


.335 


-.09^4 


.814 


.087 


16 


.880 


.326 


-.014 


, .833 


.045 




• 




^ 







Table 8 

Item .Chain Extraction for the Mathematics Datasets 



Dataset 
UMATH 



«lCrus & Bart 
Procedure 



(4-1-5-7-11-10) 

(3-2-8-15)-, 

(9-13) 

C6-16) 

(12) 

(14) 



Item Chains Extracted 



'tl 
Procedure 



(4-5-11-^12-10) 

(3-5-11-12-10) 

(1-2-12-10) 

C3- 5-14- 10) 

(9-12-10) ' 

C4-8-10) 

(1-7-10) 

(6-10) 

03-10) 

CIS) . 



Overall Consistency 
Statistics 



"t3 
Procedure 



'tl 



't3 



KR20 



V.- 



(4-3-1-2-5-6-11-12-10) 

(4-1-5-6-7-8-10) 

(9-11-12) 

(5-14-10) 

(13) 

(15) 

(16) 

(12) 



.678 .509 



.866 



AMATH (3-1-6-8-15-10) 
(4-Z-5-16) 
(9-11) 
C7-13) 
(14) 
(12) 



(3-4-6-5-7-11-12-10) 

(3-6-5-7-14-12-ia) 

(1-2-5-7-11-12-10) 

(4-6-5-8-10) 

(2-7-16-10) 

(9-14-10) 

(5-13-10>- 

05) i 



(3-4-1-2-9-6-5-7-8- 
. 15-16-11-13-14-12-10) 



.777 .764 .936 
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Note: Cutoff values of c^^^ and c^^ used were ,90 and .70 respectively, 
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or more items with' highly similar difficulty levels, all of these 
i . , , 

items will frequently not appear on the same chain. Items that are 

too close together in terms of difficulty will often fail to show 
a clear dominance relation, indicated by -a low value of McNemar's z. 
Hence, by Krus & Bart's propB^tffe^ this absence a relation wili 
pimply that the items do^ot belong to the same dimension. Correspondingly:, 
, the low £ value also means that c^^ between the^ items will also te 
very low. Thus, items similar in .difficulty often show very inconsistent 
dominance relations. . ' 

The second problem with Krus & Bart's procedure and the'c^^ procedure 
is also related to the distribution "of item difficulty leyels^ Two ^ 
items which arfe independent can show a consistent dominance relation 
which is due solely to difficulty differences between the two items. 
For example, consider two items that are independent and have difficulty * 
•levels of .30 and .90 computed from a sample of 100 persons. The 
expected number of dominances of item 1 over item 2 is equal to 
100 X p(fa^ling* item 1 & passing item = (100) (.70) (.90) = 63. 
Likewise, the expected number of dominances of item 2 ,over item 1 
ts equal to 3.* In this case, z_ - 1.39 and c^^ =« .91. This illustrates 
that items that are disj)arate in difficulty will'tend to show consistent, 
dominance relations regardless ofr-whether or not they belong to the ^ 
same factor. 

The value of c^^^for the above-mentioned example is 0. This - 
illustrates a desirable property of c^^, that the expectfed number 
of chancre* dom±i\4nces (for independent items) is taken into consider- 
ation. The* c^2 procedure is also less prone to the first problem 
<^described shove that items too similar in difficulty level tend rtot 
to show a clear dominance relation. 

The c^2 procedure yielded chains which correctly reflected the 
factor structures for all seven simulated c^ibtasets. It was found to 
he consistently superior to both the c j^^and Krus & Bart procedures. 
/The better performance of c^^ compared with c ^ is In agreement with 
results found by Cudck (1980). However, for the ^thematics data. 
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the procedure did a poor job of reproducing the factorial dimen- 
sionality. Two reasons are offered for this finding. First, the 
mathematics datasets showed a. fairly strong first factor as evidenced 
bjr^the magnitude of the first eigenvalues T The two-dimensional 
datasets showed no strong first factor • For the mathematics data^ 

unduly influenced by the first factor, which 
could have distorted the chain-extraction process. A second reason 
for the failure of c^^ to reproduce the factors for the mathematics 
data concerns the correlation between the factors. The factors tor 
the simulated datasets were all orthogonal, whereas for the mathe- 
matics data the factors were subst,antially correlated (e.g. r » .46 
for AMATH). In the case of correlated factors, the c^^ Procedure 
may not be able to distinguish between iten^ loading on different 
factors. 



Study II 

y 

An attempt was made to develop a new order-analysis procedure 
which allev. ted the problems of current procedures. Study I 
illustrated rhree major shortcomings of current order-analysis 
procedures for reproducing factorial dimensionality: 

1) Itens from the same factor with similar difficulty levels 
can be seen as being inconsistent (in the sense of showing 
about as many dominances as counter-dominances) and are 
therefore deemed to belong to different dimensions. ^ 

2) Two items f:hat are independent can show a consistent 
dominance relatiqn which is due solely to difficulty 
differences between the items. 

3) Order analysis of a set of items with an oblique factor 
structure will often not reproduce the factorial dimensions. 
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The new order-analysis procedure, termed ORDO, was designed 
specifically to address the first two of these problems. Basically, 
ORDO represents an amalgamation of Krus and Bart's (1974) procedure 
and the Reynolds (i97b) procedure using . Krus anC Bart's ap,j roach 
seemed to be a good place to start in developing a new procedure, 
as it "truly" reflects the basic ordfer-analytic principles of items 
(and persons) forming simple orders. Reynolds' procedure, on the 
other hand, deals with the consistency of an item set which is 
assumed to be an indicator of the orderability of the item set/ In 
this sense, Reynolds' approach might be termed an indirect order- 
analysis procedure, 

ORDO represents a radical departure from other order-analysis 
procedures in that it extracts partial orders of items rather than 
simple orders (see Coombs, et al., 1970). The connectedness property 
of simple orders creates the first problem with order-analysis pro- 
cedures mentioned above. Considering dimensions as partial orders 
allows for two Items to fall in the same dimension without there 
necessarily being a dominance relation between^ them. This may seem 
problematic, as the lack of a dominance relation between two items 
also represents <n:he primary evidence that those items are from 
different dimensions^ However, a pair of items from the same dimen- 
sion that do not show a dominance relation have, another characteristic — 
high proximity. The proximity measure used is the squared Euclidean 
distance between the points representing the two items^ which is 
also equal to the total number of persons for which one of the two 
items dominated the other. If two items are close together on the 
same dimension, few persons will pass only one of^them. This high 
proximity characteristic is not evident for pairs of items which do 
not measure the same dimension. 

The basic algorithm for JRDO proceeds as follows. Compute the 
item dominance matrix and reorder the rows and columns in terms of 
decreasing item difficulty level. Compute McNemar's z statistics 
for each item pair, as well as chi-square. tests for association. If 
the values of 2 and chi-square- are both significant then conclude 
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that a true relation (beyond thJt attributable to difficulty differ- 
ences) exists between the two items. If either or both are not 
significant-, then conclude that a true relation is not present. Next, 
use the relation information to extract a chain of items using Krus 
and Eart's (1974) method. This forms what is termed a "skeleton" 
chain of items. Items are then added to the chains that have high 
proximity to one of the skeleton chain members. This process 
reov^lLs in each skeleton chain member and items added to it being 
considered as an equivalence class, where items between equivalence 
classes should how consistent dominance relations, and items within 
equivalence classes should not show consistent dominance relations! 
The chain-extraction process is then repeated for items which are not 
already members of a chain until all luems are placed in a chain 
(singleton chains are allowed). The number of extracted chains is 
interpreted as the dimensionality of the dataset. 




Methut^ sad Results 

\ 

The simulated and mathematics datasets described in Study II were 
order-analyzed using ORDO. Although the results for the simulated 
data are not shown here, ORDO correctly reproduced the factors for 
all seven datasets. The results for the mathematics data are shown in 
Figures la and lb. For the UMATH datase IRDO extracted four chains, 
^o of the chains were equivalent .to the factors fp^jfid^for the two-factor 
solution given in Table 6. The ^ur chains we^ labeled: subtraction, 
addition of two negative numbers, addition of two numbers with opposite 
signs, and addition of two positive numbers! For the AMATH dataset, 
ORDO extracted two chains which were clearly the same as the two 
factors of addition and subtraction. For both datasets, chains containing 
addition items showed few equivalence classes, due to highly similar 
means for those items. 
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ORDER I : 
(Subtraction) 



4 
3 



I 

2 



5 
9 
6 



8 



ORDER g : 

( Addition of 
two negative 
numbers) 



1 1 
12 



ORDER in : 

(Addition of 
numbers with 
opposite signs) 



13 
14 
15 
16 



ORDER Ig: 

(Addition of 
two positive 
numbers) 



[10] 




Figure 1 a Ord«r analysis results for UMATH dotaset using ORIX) ( brackets denote 
equivalence classes, arrows denote dominances ). * 



ERIC 



^0 



I 



20 



ORDER I : 
(Subtraction) 



•3 
4 



I 

2 
9 



6 
5 



8 

7 



} 

ORDER n : 
(Addition) 



15 
16' 
I I 

13 
14 

12 



{,0] 



Figure lb! Order analysis results for AMATH dataset using ORDO (brackets denote 
equivalence classes, arrows denote dominances ). 
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Discussion 

The results of this study support the use of ORDO as the ord^r- 
analysls procedure to use in assessing the dimensionality of a nest* 
ORDO matched- the c^^ procedure in reproducijig the factors present 
in the simulated d^ta, and it oiaperform^d the c^^ procedure in 
determining the factor structure of the mathematics data. Apparently, 
ORDO is less sensitive than the c^^ procedure to oblique factor 
structures and/or dominant first factors in a dataset. 

The main motivation for. extracting unldimenslonal subsets of 
items concerns satisfying the unldlmenslonality requirement of latent- 
trait models. Lord & Novlck (1968) state that if performance on a 
set of items has an underlying JnultlvarJiate normal dlsttlbution and 
a single ^common factor is present in a matrix of tetrachorlc corre- 
latlon coef£iclent^, then the latent space is unldimenslonal and 
local independence holds. In this study, phi coefficients were used 
rather than tetrachorlc coefficients. There are two persistent^ 
problems with tetrachorlc- correlation coefficients. When one item 
domlna tea another item in a perfectly consistent manney (i.e., no 
counterdoSlnances) the tetrachorlc correlation is eqUkl to 1,0, 
However, since in most cases the correlation coefficient is calculated 
for sample data, one would typically be reluctant to accept 1.0 as 
a population correlation estimate. Also, matrices of sample tetrachorlc 
coeff iciepts- will often be non-Gramlan,/ln Vlolatlou of basic assiimptlona 
of the f actor- anal/tic model. Neither of these pi^pblems occur^when 
phi coefficients are used* While phi coefficients are influenced 
by* the relative difficulty levels of the items, Comrey <1973) reported 
finding the influences pf difficulty factors to be minor, and he. 
endorsed the use of phi rather^h^n tetrachorlc coefficients. 
Hence, phi coef flclentfi^^ were deemed to be appropriate in this study. 

Order analysts avoids many of the problems Involved in factor 
analysis* Also, no dlstrll)utlonal assumptions are required in 'the 
order-analytic model. This sjtudy has shown that ORDO can yield 

% 

t 
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results that are highly similar to results found with factor analysis. 
Order analysis may represent a very desirable alternative to factor 
analysis In assessing the dimensionality of tests. 

Certainly more research is necessary to determine the eventual 
usefulness of order analysis in determining item sets which are. 
appropriate* for item response theory. 
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AppeT^dix 1 

Factor Analysis Results for Dataset HI 
• *♦» * , 

Mean S,D. Factor I 

loadings 

- ' ?— 



1 


.100 


.300 


.446 


2 


.196 


.397 


.611 


3 


.312 . 


.464 


.727 


4 


.424 


.495 . 


.791. 


5 


.518 


.500 


.840 


6 


.608 


.489 


.810 


7 


.702 


.458 


.745 


8 


.810 


.393 


• .699 


9 


.914 . 


.281 


.439 


10 


.996 


.063 


.110 



Appendix 2 



Factor Analysis Results for Dataset Ml 



Item Mean S.D. Factor j. 

loadings 



1 


.330 


.471 


.711 


2 


.322 


.468 


.699 


3 


.454 


.498 


.838 


A . 


.480 


.500 


.851 


5 


.528 


.500 


.835 


6 


.566 


.496 


.845 


7 


.616 


.487 ' 


.844 


8 


.744 


.437 


.757 


9 


.736 


.441 


.753 


io 


.756 


.430 


.722 
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Appendix 3 ^ 
Factor Analysis Results for Dataset Ll^ 



Iteftt 


Mean ■ 


S.D. 


Factor I 








loadings 


1 


.480 


.500 


.866 


2 


.49.0 


.500 


.877- 


3 


.470 


.500 


.848 


^ i 


- .480 


.500 , 


.845 ' 


5 


.488 ' 


.500 * 


.846 - 


6 


.480 


.500 


.&56 


7 


.470 


■ .500 


^ .831 


8 


.474 


.500 


.854 


9 


- .4r'> 


.500 ' 


.868^ 


10 


.4i 


.499 


.856 



Appendix 4 
Factor Analysis Results for Dataset 412 



Item 


Mean 


S;D. • 


Factor I 


Factor II 








loadings 


loadings 


1 


.210 


: — ^ 

.408 


— »- ' 

-.049 


^ .591 


2 


.396 


.490 


.643 


.809 


3 


.592 


;492 


-.013 


.808 


4 


.808 


.394 


-.035 


.565 


5 


.242 


.429 


.639 


-.006 


5 


.290 


.454 


.711 


^ ..046 


7 


.458 


.499 


.827 


.011 


8 


.576 


.495 


.832 


-.057 


'9 


.716 


.451. 


.736 


-.027 


10 


.824 


.381 


.588 


-.075 
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Appendix 5 

Factor Analysis Results for Dataset M2 

Item Mean S.D. Factor I Factor II 

loadings ^ i^dings 



1 


.226 


. .419 


.083 


:680- 


2 


.416 


.493 


-.002 


.857 


3 


.436 


.496 


-.002 


.862 


A 


.718 


.450 


-.031 ^ 


.578 


5 


,122 


,328 


.667 


-.042 


6 


.106 


.308 


, .633 . 


' -.054 


7 


.400 


.490 , 


.765 . 


>-.018 ' 




.404 


.^91 . 


.762 


-.030 


9 


.904 


' .295 


.371 


.073 


10 


.90?» . 


.295 


.372 


.048 



: : ^ k 




Appendix 6 
Factor Analysis Results for Dataset L2 



Item 


Mean 


S.D. 


Factor I 


Factor II 








loadings 


loadings 


1 


.510 


,500 




,856 


2 


.500 


,500 


^6 


,832 


3 


.494 


,500 




,852 


4 


.496 


,500 




,838 


5 


.498 


,500 


.839 


,009 


6 


.522 


,500 


,848 


-,005 


7 


.522 


,500 


,819 


-,001 


8 


.512 


,500 


.848 


-,015 


9 


,510 


,500 


..JB19 


-,015 


10 


,498 


,500 


,839 


-,012 
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Appendix 7 
Factor Analysis Results for Dataset MIO 



Item Mean S.D. factor Eigenvalue % of 









Number 




• Variance 


1 


.072 
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11.7 


3 


.368 ' 


.483 


3 


l.U 


11.1 


4 


.414 


.493 


4 


1.07 
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9.5 
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.744 


.437 
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0.87 
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.824 


.381 
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10 


.898 
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10 
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